upgrading to spark 2.2.0 #4314
Merged
upgrading to spark 2.2.0 #4314
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lbergelson
force-pushed
the
lb_update_spark_2.2.0
branch
2 times, most recently
from
002687c to
23066ea
Compare
|
This is exploding due to netty conflicts between version 4.0.4x and 4.1.x. I'm not sure why the move to spark 2.2.0 changed this exactly, but the version that causes problem is imported by google's gatk-java library. Removing that removes the conflict. I rebased this ontop of the branch I have in PR that removes that library and hopefully it won't fail... |
Codecov Report
@@ Coverage Diff @@
## master #4314 +/- ##
===============================================
- Coverage 79.024% 79.017% -0.007%
+ Complexity 16447 16440 -7
===============================================
Files 1047 1047
Lines 59186 59186
Branches 9672 9672
===============================================
- Hits 46771 46767 -4
- Misses 8659 8661 +2
- Partials 3756 3758 +2
|
this bumped some transitive dependencies which required a minor update in unrelated classes
lbergelson
force-pushed
the
lb_update_spark_2.2.0
branch
from
23066ea to
7bd7707
Compare
|
@lbergelson How's this one looking? Can it be merged yet? |
|
@lbergelson Are you comfortable merging this one yet? |
|
@lbergelson As part of your current set of Spark test runs, can you also test whether this PR appears to work on dataproc? |
cwhelan
pushed a commit
to cwhelan/gatk-linked-reads
that referenced
this pull request
May 25, 2018
* upgrading to spark 2.2.0 this bumped some transitive dependencies which required a minor update in unrelated classes * updating hadoop version to match the version in dataproc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This bumped some transitive dependencies which required a minor update in unrelated classes
We shouldn't merge this until we get a 👍 from the SV team as well as running the jenkins spark tests.
I think the SV team is already using 2.2.0 since they've gone to dataproc image 1.2.
This will prevent the annoying adam log spam, closes #4186
closes #2555